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Abstract 


This research is aimed to validate an instrument, the Malaysian version of ROSE or MROSE to 
gauge Malaysian secondary students’ interests, attitudes, values, and priorities in S&T-related 
issues. Partial Least Squares-Structural Equation Modeling approach was used to evaluate the 
validity and reliability of the instrument. The internal consistency reliability (composite reliability 
and Cronbachs Alpha coefficient), convergent validity (Average Variance Extracted), and 
discriminant validity (cross loadings, Fornell-Larcker criterion, and Heterotrait-Monotrait ratio) 
for each individual item of the instrument were being assessed. 

Keywords: affective factor, PLS-SEM, science and technology, relevance of science education 
(ROSE). 


Background 


The Relevance of Science Education (ROSE) is a cooperative research project 
with wide international participation, addressing mainly the affective dimensions of how 
young learners relate to Science and Technology (S&T). The purpose of ROSE is to 
gather and analyze information from the learners about several the factors that have a 
bearing on their attitudes and motivation to learn S&T. These include a variety of S&T- 
related out-of-school experiences, interests in learning various S&T topics in different 
contexts, prior experiences with and views on school science, views and attitudes to 
science as well as scientists in society, future hopes, priorities and aspirations as well as 


young people’s feeling of empowerment with regards to environmental challenges, etc. | 
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Problem Statement 


Through international deliberations, workshops and pilot studies among many 
research partners, ROSE has developed an instrument that aims to map out attitudinal 
or affective perspectives on S&T in education and in society as seen by 15- year old 
learners. The ROSE advisory group comprises key international science educators from 
all continents. They have tried to make an instrument that can be used in widely different 
cultures. The aim is to stimulate research cooperation and networking across cultural 
barriers as well as to promote an informed discussion on how to make science education 
more relevant, also meaningful for learners in ways that respect gender differences and 
cultural diversity. The ROSE international science educators also hope to shed light on 
how they can stimulate the students’ interest in choosing S&T-related studies and careers 
as well as to stimulate their life-long interest in and respect for S&T as part of their 
common culture. To ensure the ROSE instrument is free from gender and cultural bias in 
Malaysian context, it is extremely crucial to validate a Malaysian version of Relevance of 
Science Education (MROSE) Questionnaire as a valid and reliable measure of Malaysian 
secondary students’ interests, attitudes, values, and priorities in S&T-related issues. 


Research Aim 


This research embarks on a research objective to validate a Malaysian Relevance 
of Science Education (MROSE) Questionnaire to gauge Malaysian secondary students’ 
interests, attitudes, values, and priorities in issues related to science and technology. 


Research Methodology 
Population 


The Relevance of Science Education (ROSE) target population is the cohort of 
all 15-year-old students, or the grade level where most 15-year old students are likely to 
go. In Malaysia, this corresponds to the last year attendance of lower secondary school 
(Form 3 or Grade 8). However, this cohort of students is not accessible to researchers as 
they will be sitting for a lower secondary examination, or in Malay language Pentaksiran 
Tingkatan Tiga (PT3), and the Ministry of Education will deny researcher permission to 
conduct research on them. For this reason, the Malaysian ROSE target population has to 
be altered to the cohort of students of 16- year old who have just entered Form 4 (Grade 
9) and the survey was conducted at the early part of the school year. As ROSE samples 
school classes and not individual students, the Malaysian accessible population is more 
precisely defined as the classes at the early stage of Form 4 where most 16-year old 
students are likely to go. 


Sample and Participation 
A list of all secondary schools in Sabah and their relevant statistics for 2018 was 


obtained from the Sabah State Education Department. The samples of this study were 
drawn from the list using a stratified sampling routine. The first sampling unit is the 
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school. Once the schools were selected, the Form 4 classes in these schools were made 
the targets of a second random sampling routine. The science teachers who have been 
appointed as the research assistants in the selected secondary school have carried out the 
random sampling routine. Once the classes were chosen, the students in these classes 
formed the samples of this study. 

Due of budget limitation, the research team had adopted a stratified sampling 
strategy based on geographical region in Sabah, i.e. West Coast Division, Interior 
Division, Kudat Division, Sandakan Division, and Tawau Division. Using a computer 
random generator, one school was drawn from each division of Sabah. The resulting 
sample of 5 schools is expected to possess the essential state characteristics. 

Official letters were sent to the sampled schools inviting them to participate in 
the revised ROSE survey. A description of the ROSE project together with copies of 
the permission letter from the Educational Planning and Research Division, Ministry 
of Education and the Sabah State Education Department were attached. The school 
principals were asked to enclose statistics on the number of Form 4 classes by stream 
(track) and the number of students in each class for further sampling purpose. For 
practical purposes, the sampling of the classes was performed by the research assistant in 
the selected school. Specific instruction was given to the research assistants to randomly 
select two science classes and two non-science classes that were involved in this study. 


Instrument 


Student Background Questionnaire (cover sheet) 
The cover sheet of the MROSE questionnaire contains five classification questions: age, 
gender, region, school location, track (science or non-science). 


A. “What I Want to Learn About” (48 items) 

This dimension of questions will give empirical evidence on what topics various 
groups of students are interested in learning about. This insight can inform our 
discussions on how S&T curricula can be constructed in order to meet the interests 
of different groups of learners. It should be emphasized that the prime concern 
behind this question is that science lessons should engage the students. Asking 
the students how interested they are in various topics is one approach for getting 
in touch with science lessons’ potential for engagement. However, engagement 
does not refer simply to enthusiasm, entertainment and fun. It is also important 
to trigger concern, provoke creative thinking and stimulate individual growth. 


B. “My Future Job” (26 items) 
This question provides information about the future priorities and motivations of 
the students. This is in itself interesting information and allows for comparisons 
across cultures and between various groups of students. 


C. “What I Want to Learn About” (18 items) 
Under this heading, the following instructions are given: How interested are you 
in learning about the following? (Give your answer with a tick on each line. If 
you do not understand, leave the line blank). 
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D. “Me and the Environmental Challenges” (18 items) 

Empowering students to deal responsibly with the environmental issue should 
be an important goal of education. As science educators we need to develop 
knowledge and awareness of what challenges we are facing in our efforts to 
make students equipped to meet the environmental problems. Research in 
science education have taught us a lot about students’ conceptual understandings 
(and ‘misconceptions’ or ‘alternative conceptions’) of science contents, but less 
about their attitudes, priorities, and decision-making regarding environmental 
matters. This part of the questionnaire will deepen our understanding of how 
youth relate to some environmental issues, and the results against perspectives 
from sociology and youth research will be interpreted. 


E. “What I Want to Learn About” (42 items) 
Under this heading, the following instructions are given: How interested are you 
in learning about the following? (Give your answer with a tick on each line. If 
you do not understand, leave the line blank). 


F. “My Science Classes” (16 items) 
Under this heading, the following instructions are given: Jo what extent do you 
agree with the following statements about the science that you may have had at 
school? (Give your answer with a tick on each line. If you do not understand, 
leave the line blank) 


G. “My Opinions about Science and Technology” (16 items) 
Under this heading, the following instructions are given: Jo what extent do you 
agree with the following statements? (Give your answer with a tick on each line. 
If you do not understand, leave the line blank) 


H. “My Out-of-School Experiences” (61 items) 
Under this heading, the following instructions are given: How often have you 
done this outside school? (Give your answer with a tick on each line. If you do 
not understand, leave the line blank) 


Data Analysis Procedures 


Partial Least Squares-Structural Equation Modeling (PLS-SEM) approach was 
used to evaluate the validity and reliability of the MROSE questionnaire. Before running 
PLS-SEM analysis, the data collected were screened to ensure error free from missing 
value, suspicious response patterns, and outliers. Results from the statistical analysis were 
being reviewed and evaluated in terms of the relation among items in the measurement 
model. To ascertain the validity and reliability of the MROSE questionnaire, the 
internal consistency reliability, convergent validity, and discriminant validity of each 
individual item were being assessed. In particular, internal consistency reliability for 
each subscale was determined through the composite reliability (CR) and Cronbach’s 
Alpha (CA) coefficient. The Average Variance Extracted (AVE) was evaluated to assess 
the convergent validity of the instrument. Cross-loadings, Fornell-Larcker criterion, and 

| Heterotrait-Monotrait ratio (HTMT) were also assessed to evaluate the discriminant 
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validity for each item in the MROSE instrument. 


Research Results 


The ultimate goal of this study is to examine the validity and reliability of a 
Malaysian Relevance of Science Education (MROSE) questionnaire by using PLS- 
SEM approach. Reflective and formative measurement models have been developed and 
evaluated to assess the validity and reliability of the MROSE questionnaire using PLS- 
SEM approach. Figure 1 depicted the measurement models of MROSE questionnaire 
after item deletion. The measurement models consist of five latent constructs which 
are ‘Me and the Environmental Challenges (MEC)’, ‘My Out-of-school Experiences 
(OOSE)’, ‘My Science Classes’ (MSC). ‘My Opinion about Science and Technology 
(MOST)’, and ‘My Future Job (MFJ)’. 


The Assessment of Reflective Measurement Models 


In assessing the reflective measurement models of the MROSE questionnaire, the 
internal consistency reliability and the discriminant validity of the measurement models 
are evaluated. According to Hair, Hult, Ringle, and Sarstedt (2017), the threshold value 
of the composite reliability (CR) ranges from .70 to .95 for an exploratory research, 
whereas the threshold value for Average Variance Extracted (AVE) is above .50. 

Table 2 showed the Cronbach’s alpha, CR and AVE values for all the three 
reflectively measured latent constructs (i.e, MEC, MOST, and MSC) before item 
deletion. The Cronbach’s alpha for MEC, MOST, and MSC before item deletion are .540, 
.825, and .915, respectively. The CR values for MEC, MOST, and MSC are reported as 
.465, .866, and .923, respectively. All the three constructs reported AVE values less than 
50, (.e., .145, .366, and .443, respectively). Hence, the AVE values for all the three 
reflectively measured constructs did not meet the minimum threshold value of .50 as 
recommended by Hair et al. (2017). Therefore, all items with outer loadings range from 
.40 to .70 have been considered for item deletion, whereas items with outer loadings less 
than .40 have been eliminated from the measurement models. 

Table 3 presents the deleted items based on the item outer loading assessment of 
the reflective measurement models. For the MSC construct, four items (Fl, F3, F14, and 
F16) have been deleted from the measurement model which have led to the improvement 
of the outer loadings for the remaining items and also the AVE for each construct. As 
for the MOST construct, out of 16 items, seven items (G6, G7, G8, G9, G10, G12, G14) 
have been deleted. Out of 18 items in measuring the MEC construct, a total of 15 items 
(D1, D2, D3, D4, D5, D6, D8, D9, D11, D13, D14, D15, D16, D17, and D18) have been 
deleted due to low outer loadings. 

After all items which have not met the minimum requirement of outer loadings 
as recommended in PLS-SEM (Hair et al., 2017), the validity and reliability of the 
reflectively measured constructs were reexamined and presented in Table 4. Based on the 
results shown in Table 4, the Cronbach’s alpha, CR, and AVE for all the three reflectively 
measured constructs have shown high reliability. The Cronbach’s alpha for MEC, MOST, 
and MSC is reported as .669, .873, and .916, respectively. The CR for MEC, MOST, and 
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MSC is reported as .802, .899, and .925, respectively. The AVE values for all the three 
reflectively measured constructs meet the minimum threshold value of .50. 

On the other hand, the measurement models of the MROSE questionnaire have 
been assessed in terms of its discriminant validity through the heterotrait-monotrait ratio 
(HTMT). Based on the results shown in Table 5, the HTMT values for all the reflectively 
measured latent constructs are less than the threshold value of .85 which indicates that 
the constructs have high discriminant validity after item deletion. 


The Assessment of Formative Measurement Models 


As for the assessment of formative measurement models of the MROSE 
questionnaire, the collinearity issue has been checked via the variance inflation factor 
(VIF) values. Following this, the assessment of significance and relevance of the 
formative indicators were performed by means of bootstrapping procedure. According to 
Hair et al. (2017), formative indicators with VIF values exceed the minimum threshold 
of 5.0 need to be deleted. Appendix A presents the VIF values for all the items measuring 
MFJ and OOSE constructs after item deletion is made. It is clearly shown that all the 
items measuring MFJ and OOSE possessed VIF values less than 5.0. This indicates 
that both formatively measured constructs have no collinearity issue. Table 6 shows the 
assessment results for collinearity issue via VIF. All items measuring MFJ remained 
however eleven items (H1I1, H15, H16, H21, H34, H36, H46, H49, H56, H57, and 
H59) measuring OOSE have been deleted due to the VIF values exceeded the minimum 
threshold value of 5.0 (Hair et al., 2017). 

Following the assessment of collinearity issue is the evaluation of the significance 
and relevance of the formative indicators by means of bootstrapping procedure. The 
bootstrapping results (Appendix B) showed the p values and the outer weight (OW) of 
all the formatively measured items. According to Hair et al. (2017), formative indicator 
which is non-significant (p > .05) with low outer weight below .10 must be deleted for 
further analysis. A total of 15 items measuring MFJ construct and 43 items measuring 
OOSE construct have been deleted from its respective formation measurement model as 
these items were not significant and with outer weights less than .01 (Table 7). 


Conclusions 


The assessment of reflective and formative measurement models in the MROSE 
questionnaire has shown that MROSE questionnaire is a valid and reliable instrument 
to measure Malaysian secondary students’ interests, attitudes, values, and priorities in 
issues related to science and technology after item deletion is made. The remaining 
items in the reflective measurement models of the MROSE questionnaire have met the 
minimum threshold values as required by outer loading, Cronbach’s alpha, composite 
reliability, average variance extracted, and heterotrait-monotrait ratio. On the other hand, 
the remaining items in the formative measurement models of the MROSE questionnaire 
have met the minimum threshold values of variance inflation factor and the significance 


| 


Proceedings of the 3“ International Baltic Symposium on Science and Technology Education, BalticSTE2019 | 


62 


and relevance assessment. 
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